Table-projected music software with physical interaction. 

- Extending the concept of keyframe interpolation into the domain of music 
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Abstract 

My project aims to structure and simplify the 
heterogenic, fragmented style of traditional music- 
sequencer software, which relies heavily on modes and 
sub-modes to perform various tasks. 

I have applied two strategies. The first experiments 
with the physicality in the human/ machine interaction. 
The second explores geometric information 
visualization. 

Using a single graphical object to visualize control- 
parameter-settings opens up for time-based 
transformations, extending the concept of keyframe 
interpolation from animation art into the domain of 
music. This approach offers the user a lot of creative 
feedback. 
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introduction 

I have built a table-based soundfile editor, which 
utilizes various physical objects for sound manipulation 
and organizational task solving. (Figure 1) 




(Figure 1) 



The GUI is utilizing one single holistic graphical 
object for controlling relevant parameters. This gives 
the user a better overview of current ongoing musical 
events. The representation deals with isolation problems 
between individual sounds and thereby opening up for 
anticipation and comparison in the listening process. 
By taking a few contrasting snapshots of musical 
situations, represented by holistic graphical objects, the 
user may interpolate between the situations in time. 



This simple, but powerful technique gives a tool to 
create contrasts and consistent related variations, 
qualities which are fundamental and attractive for all 
time-based art. 

My approach is partly motivated in frustration with 
existing software, and partly by two main principles in 
my own artistic working method. When developing a 
"new" synthesis model, I often use random generation 
of parameter-settings, combined with interpolation 
between the saved settings (presets), to search for 
interesting sounds. I find this to be a fruitful way to 
generate new related sounds. 

A similar way of thought has been introduced 
by J. C. Risset [9] through the method of analysis by 
synthesis. Through synthetic imitation of a sound one 
may learn something about the sounding object by 
determining the most important parameters for 
imitation. Through imitation and effective 
representation one may also create interesting variants 
as by-products. 

Holistic representation 

The musical object is a well established metaphor in 
musicology, and has been a way of dividing the stream 
of sound in art music. The idea of the musical object is 
more adapt to the textural focus of contemporary music, 
and studio-based popular music. Its a"... multi- 
dimensional entity having a number of dimensions and 
sub-dimensions such as harmonic content, pattern of 
change or fluctuations in harmonic content etc" 
(R.I.God0y, unfinished manuscript. University of Oslo, 
Dept. of musicology.) 




(Figure 2) [10] 



The complex nature of timbre can effectively be 
visualized by means of multi-dimensional 
representation. (Figure 2) This is a way of visualizing 
various aspects of an entity in an n-dimensional 
conceptual space. This representation visualizes a broad 
range of qualities, and offers an environment well suited 
for system comparisons, and scaling. 

Implementation 

I wanted to use a similar method for visualizing the 
most important control -parameters of various sound 
effects. Still it has proved more fruitful to work in two, 
rather than three dimensions, especially when dealing 
with time animations. [3] Deceptions of perspective 
often confuses the height and depth. I chose a similar 2- 
dimensional model introduced in design analysis with 
examples of comparisons of various music-systems by 
Birnbaum [2]. In this model several axis grow out from 
an origo-point. Each axis consists of a set of contrasting 
qualities. (Figure 3) 




(Figure 3) [2] 



This proved to be a fruitful model for the 2-dimensional 
table surface. I chose to let each "quality" be a tentacle 
on a central graphical object. The objects represent a 
sounds with indication of strength, location in space, 
location in time and various effect-settings 




(Figure 4) 



Each effect, and each of its parameters consist of two 
contrasting settings, which may be edited (red/green 
mode). The two settings can be interpolated (orange 
mode) (Figure 4) to create new consistently varied 
musical objects. I have tried to clarify this through 
symbolic coding. Aesthetically the red and green 
represents the contrasts, and the orange represent the 
consistent variations in between. 

Geometrical shapes are well suited for 
information visualization. They can transmit a lot of 



information at a glance and relations between objects 
are still recognizable despite minor changes, and are 
thereby robust to deformation. We know also from 
popular experiences of memory mind maps, that simple 
geometrical shapes enhances our ability of short term 
memory greatly. 

Physical interaction 




(Figure 5 & 6) 



Prototype-scenario 



I am hoping to further simplify the user experience by 
making metaphorical projections from Image schemata 
into the physical domain. This is a direction in cognitive 
psychology stating that we organize our impressions 
according to patterns, which is founded in the 
experience of our own bodies in interaction with the 
world. Future possibilities lies in 

a) turning old computer metaphors into the 
physical domain, e.g. moving physical objects 
inside graphical objects to edit various 
soundqualities, instead of e.g. putting virtual 
documents inside virtual folders 

Currently I use a multi-sensory resistor-based modal 
cube to access a few simple unified trinity modes: Play- 
freeze-stop, add-activate-delete, contrast-I - interpolate 
-contrast-II. 

In addition I use a tracking object for positional 
movement, and rotational movement. This could be 
further extended to an object, which may be turned or 
filled. (Figure 5) 

b) Making metaphorical projections of "unused" 
image schemata into the virtual domain, e.g. 
the metaphor of objects moving on a path from 
point a to b. 

Building on this idea I let the soundfiles be represented 
with graphical objects moving on paths in time (Figure 
7), where the length of the line is the length of the 
soundfile and changes in control-parameters are 
visualized with continuous animated feedback. 




(Figure 7) 



Modes of interaction 

The interface have three main modes of interaction. 

• Distribution of soundfiles in the time-domain 

• Editing (adding and deleting) presets fixed to a 
certain soundfile at a certain point in time. 

• Playback of a session of saved parameter- 
settings with visual and auditory interpolation 
between presets. 

Part 2 and 3 are currently realized. 

By placing physical objects (PhO) on top of the 
graphical object (GO), PhO will be able to perform 
different editing tasks. 

- Movement Mode . Volume (y-axis) "Panning" 
(rotation), time displacements (x-axis) 

- Orange Mode (interpolation mode) is a high level 
control, where the various parameters for each effect is 
controlled by the alteration of one single graphical 
feature (tentacle). 

By moving the PhO into one of the tentacles it may alter 
the relative mix between the different effects (x/ y axis), 
and interpolate between two extreme parameter-settings 
for the equivalent effect, by means of rotation 

- Green and red Mode (contrast modes) By using a 
random generator one may generate interesting 
contrasting parameter-settings within the same effect. 
(These may be saved in the green and red mode.) This is 
a lower level of control, where all parameters are 
graphically available, both for random manipulation and 
manual fine-tuning. When the two extreme settings are 
fixed, one may interpolate between the two settings in 
the orange mode. When the user is satisfied with a 
certain object- setting, she may save it as a snapshot 
The playback mode (Mode 3) will consist of different 
intersecting lines with objects moving on lines. (Figure 
7) When playback is initiated the GOs start to move 
with the cursor until they reach the end of their line. 
The length of the lines represent the length of the 
soundfile, and the vertical direction is the volume. The 
GOs move on the lines and change continuously 
according to the interpolated values between time- 
presets (snapshots) containing information about 
volume, panning, effectmix (tentacle-size) and high- 
level control-interpolation (tentacle shape ) 

This multi-sensory environment (visual and 
auditory) gives the user a good overview of the current 
musical situation, and the ability for prediction of future 
events. By using colour or textural coding of tentacles 
the user would get a much more intuitive overview of 
ongoing events. An experienced user will start to 
recognize the objects as a holistic object, following the 
outlines of the tentacles, not having to perceive each 
tentacle individually. 

Conclusion 

I have described a table-based music 
sequencer/controller with physical interaction, which 
introduces keyframe interpolation into the domain of 



music, offering the user more creative feedback. It is 
simplified through holistic visual representation of 
sound but should nevertheless include most of the major 
features of a soundfile - sequencer, and thereby have 
the potential to be more than a toy for sounds. 
Existing software are either dealing solely with the 
intuitive style of real-time editing and improvisation e.g. 
Audiopad , or with the accuracy of playback of saved 
settings. My interface is aiming to unify these aspects 
through the concept of musical keyframe interpolation. 
Technology is widly availible today for a low cost, 
music software included. New users are overwhelmed 
by the technical possibilities and the fashyness of 
technology. By giving the user the task of choosing just 
a few contrasting settings she is offered a lot of creative 
feedback from the system. 

Physical and technical configuration 

Graphics and physical/virtual interaction in 
PROCESSING [8] , Sound and data-storage in 
MAX/MSP. [6] 
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